Finding and Identifying Text in 900+ Languages

This paper presents a trainable open-source utility to extract text from arbitrary data files and disk images which uses language models to automatically detect character encodings prior to extracting strings and for automatic language identification and filtering of non-textual strings after extraction. With a test set containing 923 languages, consisting of strings of at most 65 characters, an overall language identification error rate of less than 0.4% is achieved. False-alarm rates on randomdataare0.34%whenfiltering thresholds are set forhigh recall and0.012%when set for high precision, with corresponding miss rates of 0.002% and 0.009% in running text. a 2012 R.D. Brown. Published by Elsevier Ltd. All rights reserved.